A frequent and powerful use of a processor-controlled machine such as a computer is the presentation of information in the form of images on a display device connected to the machine. An image, which may include characters, words, and text as well as other display features such as graphics, is produced in a display area of the display device directly from an image definition data structure defining the image; the image definition data structure is typically stored in a memory area of the machine. One type of image format known as a raster image is composed of individual image locations commonly referred to as "pixels". The discussion of images herein will generally reference pixel data, but it is to be understood that other data formats, such as vector data, may also be used to define an image. The images discussed herein may be either static (having the appearance to a system operator or user of not moving), or animated, as, for example, in the case of a digital video image.
An image definition data structure alone carries relatively limited data about the information content of the image the data represent. However, for many images, the image definition data structure is itself generated from another data structure which contains information or data capable of being understood by a human or by a software operation such as an application program which is executed by the machine's processor. Such a data structure will be referred to herein as the "information model data structure" or the "model data structure", and is to be distinguished from the image definition data structure.
An example of an information model data structure is a model from which a graphics image, such as a photorealistic scene, is generated. The model, also called a scene description, contains descriptions of primitives that define the shapes of components, objects, and light sources in a three-dimensional scene, the displayed appearance attributes of the primitives, such as color and surface texture, and connectivity relationships that show how object primitives are positioned and fit together in the scene. The scene description must then be converted into the image definition data structure, such as a pixel data structure, for display of the image on an output medium, typically on a monitor or in hardcopy form. The operation, or application, of mapping the scene description to an image definition data structure is typically referred to as "rendering".
Another example of an information model data structure is a document data structure, or document model, used by a word processing or other document manipulation application which contains the text, formatting instructions, and other data needed for producing a formatted document. In order to permit interaction with the text and its format on a page or in a document by an operator or user of the processor-controlled machine on which the application instructions are executed, an application such as a word processing program produces an image definition data structure from the document data structure (the model) in order to display an image of the page or document in the display area of the display.
Still another example of an information model data structure is a graphical object data structure of the type used by a "illustration" or "drawing" application which contains data about each graphical object defined for display. The data about each graphical object typically includes appearance attribute data including the size, shape, color, line thickness, orientation, and the object's position in some relative coordinate system, which together describe how the object appears in an image created from the graphical object data structure. In software applications which permit manipulation of graphical objects, images may be created in which objects have the appearance of being in different planes in the two-dimensional display area, resulting in what is known as a "21/2" dimensional (21/2D or 2.5D) image. The graphical object data structure which provides description data for a 2.5D image also includes a list of the objects in an order which describes the relative positioning of the objects in the different planes and with respect to each other.
The examples just discussed are by no means exhaustive of the examples of model data structures from which images may be produced. Images produced from model data structures have at least one similarity of importance to the discussion of the invention herein: each image provides a view of the model data structure to the machine operator or user. The content of the view is established by the functions defined in the processor-controlled operation, typically called the "application", which produces the image from the model data structure. In effect, the image produced by the application provides visual access to the data and information in the model data structure.
In some existing systems, the system user may access, and in some cases manipulate, data and information in the model data structure through alternate functions defined by the application. This provides the system user with the ability to access, and perhaps affect or manipulate, data and information that is not represented by display features currently visible in the original image.
For example, in some computer-aided design systems, at the command of the user, an image including a front view (the first image) of a part assembly is displayed in the display area of the system's display device, and the user may cause a side view (the second image) of the part assembly, or a portion of the part assembly, to be displayed simultaneously with the first image in a separate portion of the display area. The second image may be displayed at one side of the display area so as to permit viewing and inspection of, and perhaps interaction with, both images. The image portion containing the front view of the part assembly is produced, typically in response to a user command, from accessing and operating on the information model data structure containing all of the part assembly information, and then a first image definition data structure is produced for displaying the first (front view) image. Similarly, in response to a user command to display the side view of the same part assembly, the information model data structure for the part assembly is again accessed and operated on to select the part assembly information needed to produce the side view, and then a second image pixel data structure is produced for displaying the second (side view) image. In some system implementations, the system designer may have decided that the second image is of primary importance to the user and so it may be displayed on top of the first image, obscuring all or part of the first image from the user's view.
Another example of the display of related simultaneous first and second images is a computer spreadsheet application that provides the ability for the user to select a cell of the spreadsheet which contains summary data in order to produce a second image containing the detailed information that was processed to produce the summary information in the selected cell. The operation to produce the second image must operate on the model for the spreadsheet to select and format the necessary data to display in the second image. The detailed information in the second image is typically displayed in a separate workspace and may obscure a portion of the summary spreadsheet of the first image.
In each of the examples described, and typical of many of the systems implementing these types of multiple views of a model, the spatial relationship between the original image, the specification of the input content for the second image and the position of the second image in the display is not easy to visualize. While the specification of the input may be directly related to the first image (i.e. when the user can point to or select a region in the first image) this is not always the case as, for example, when the user has to type commands to specify a second image. Furthermore, once a selection is specified, the user either has no control over the location of the second image, or must reposition it in a separate step. Therefore, it can be difficult to understand how the second image is spatially related to the first image and to the model. Linking direct selection in the first image to the spatial position of the second image can make these relationships more clear. This can be especially important when the model is complex and can be viewed in many different ways. As systems supporting complex models, or sets of related models, are becoming more common, this problem is becoming more critical.
An example of providing a user with simultaneous control over both first image portion selection and second image display location is the software implementation of image magnification. U.S. Pat. No. 4,800,379 discloses a method and an apparatus for displaying an image, defined by digital data representing the color content of the pixels of the image, which is responsive to a "magnify" signal from an indicator assembly of the apparatus to cause the monitor to display, centered within the boundary of an image selecting outline area, the magnified portion of the image which falls within the boundary of the image selecting outline area. The magnified portion of the image can thus be seen in the spatial context of the entire image, with the portion of the image under the magnified portion in the outline area being suppressed. The portion of the image for which the second, magnified view is desired is defined by the location in the image to which the user moves the image selecting outline area. In the magnifier disclosed in U.S. Pat. No. 4,800,379, it appears that a separate command is required for the user to initiate the magnify operation itself (see col. 3, lines 58-65).
EPO 0 544 509, entitled "Photographic Filter Metaphor for Control of Digital Image Processing Software", discloses another example of the display of a second image in the spatial context of a first image. EPO 0 544 509 discloses a method and interactive user interface for displaying an image on a display device, selecting a filter for overlaying on a portion of the image, modifying for display the portion of the image overlayed by the filter while preserving the image as it appeared before modification, displaying the modified filtered portion of the image and retrieving the image as it appeared before the modification. Predetermined or custom designed filter functions may be moved over an image and over each other to produce a modified visual display showing the effect of the filter or filters on the image without altering the actual image. Examples of filters include gamma, convolution, mixer, halftone, rotation, scaling and geometric filters which provide a variety of digital image processing effects.
Operations on the image pixel data structure alone, such as those disclosed in U.S. Pat. No. 4,800,379 and EPO 0 544 509, provide versions and views of the first image which are limited strictly to manipulations of pixel values. So, while a typical software implementation of a pixel magnifier provides some desirable features for generating the simultaneous display of two images, the range of information content possible in a second image which is derived directly from the image pixel data structure of the first image is necessarily limited to versions and views of the first image which result only from operating on the pixel values. When information is available about the first image from the information model data structure of the first image, the method used for operating on the image pixel data structure to produce image (pixel) magnification is not transferable to operating on the information model data structure to provide alternate views of the first image, and as a result, such a method is inoperable for use in accessing alternative views of an information model data structure.